Recent years have seen increasing demands for measuring the distance from a predetermined position to an object or the three dimensional position of the object with high accuracy. For example, for a case of supporting a field of view from a car, one method provided is intended to more reliably prevent a collision between the car and surrounding objects by simultaneously displaying not only a video of the surrounding objects captured by a camera attached to the car but also displaying more accurate distance information indicating the distances between the car and the objects measured by a distance measuring apparatus. In addition, for a case of a mobile phone or a television set, an input/output device for three dimensional video is desired to more faithfully reproduce such video that provides a sense of reality. For these applications, there is a demand for increasing accuracy in distance measurement by distance measuring apparatuses.
Conventionally used schemes for measuring a distance to an object or the three dimensional position of the object include stereo distance measuring schemes applying a triangulation principle. According to such a stereo distance measuring scheme, the distance to the object is calculated based on a parallax that occurs between plural cameras.
FIG. 1 is a diagram illustrating an example of calculation of a distance to an object according to a stereo distance measuring scheme in the case of using two cameras that are a camera a and a camera b. A light beam 101a forms an image on an imaging area 104a through an optical center 105a of a lens 102a of the camera a, and a light beam 101b forms an image on an imaging area 104b through an optical center 105b of a lens 102b of the camera b. Optical axes 103a and 103b represent the optical axes of the cameras. Here is an exemplary case where (i) an image of the object 100 is formed at an image forming position 107a distant, by Pa, from an to intersection 106a between an imaging area 104a and the optical axis 103a in the camera a and (ii) an image of the object 100 is formed at an image forming position 107b distant, by Pb, from an intersection 106b between an imaging area 104b and the optical axis 103b in the camera b. In this case, a parallax P (=Pb−Pa) occurs between the is camera a and the camera b. The parallax P changes depending on a distance D between the distance measuring apparatus and the object. The optical axis 103a of the camera a and the optical axis 103b of the camera b are parallel to each other. When the interval is denoted as a base line length B, and the focal lengths of the cameras a and b are denoted as f, the distance D to the object is represented by Expression (1). Accordingly, assuming that the base line length B and focal lengths f are already known by a previously performed calibration, it is possible to calculate the distance D to the object 100 by calculating the parallax P.
                    [                  Math          .                                          ⁢          1                ]                                                            D        =                  f          ⁢                                          ⁢                      B            P                                              (                  Expression          ⁢                                          ⁢          1                )            
It is to be noted that the optical axes of the cameras a and b are not parallel in real environments in most cases. In such cases, there is a need to perform rectification as shown in Non-patent Literature 1, for example. As a result, an image obtainable in the case where the optical axes of the lenses are parallel is generated, thereby making it possible to calculate the distance D according to the calculation of Expression (1).
Each of the imaging areas 104a and 104b are generally configured with imaging devices such as CCDs (Charge Coupled Devices) and CMOS (Complementary Metal Oxide Semiconductors). Accordingly, the parallax P is calculated using luminance signals from the object image discrete on a two dimensional plane, and the parallax detection resolution corresponds to normally one pixel. The distance measurement resolution (hereinafter referred to as distance measurement accuracy) is determined based on the parallax detection resolution according to Expression (1).
A description is given of an exemplary method of detecting a parallax P performed by a conventional parallax detecting apparatus. The parallax detecting apparatus calculates SADs (Sums of Absolute Differences) for an image a captured in the imaging area 104a of the camera a and an image b captured in the imaging area 104b of the camera b. A SAD is a correlation value between each of small areas in an image and a corresponding one of small areas in a corresponding image. Next, the parallax detecting apparatus calculates, using the calculated correlation value, the shifts between the sub-images of pairs of mutually corresponding small areas in the images a and b. It is to be noted that SADs are examples of correlation values, and other available correlation values include generally-known SSDs (Sum of Squared Differences) and NCC (Normalized Cross-correlation).
The following describes calculation of a parallax using SADs that are representative correlation values, with reference to FIG. 2 to FIG. 5.
FIG. 2 is a diagram illustrating a method of representing luminance of each pixel in an image. As shown in FIG. 2, “0” denotes black, “15” denotes white, and a luminance level of each pixel is represented by a line density. A luminance may be a value at or below the decimal point.
FIG. 3A is a diagram showing a part of the image a when the texture of the object in the image is viewed from the object side. FIG. 3B is a diagram showing a part of the image b when the texture of the object in the image is viewed from the object side.
An image block 201b enclosed by bold lines in FIG. 3B forms the same image as an image of the image block 201a enclosed by bold lines in FIG. 3A in the case where the object is present at an infinite distance. In the case where the object is present at a finite distance, a parallax occurs as shown in FIG. 3A and FIG. 3B, and thus the image of FIG. 3B is formed with a shift to the right compared with the image of FIG. 3A. Here is given a description in the case where there is an actual parallax corresponding to 3. 6 pixels between FIG. 3A and FIG. 3B. In order to search the image region having the highest correlation with the image block 201a, the parallax detecting apparatus shifts, by one pixel, the image block 201b to the right from the position shown by the bold lines in the FIG. 3B, and calculates the SAD for each shift amount according to Expression (2).[Math. 2]SAD=Σ|Ia(i,j)−Ib(i,j)|  (Expression 2)
Here, each of Ia and Ib denotes a luminance value within each image block, and each of i and j denotes a local address within the image block. The image block 201a and the image block 201b have the same image size. The parallax detecting apparatus calculates, for each shift amount, the total sum within the block of the absolute values of differences in luminance in the same addresses of the pairs of mutually corresponding image blocks. The image blocks may have a rectangular shape or a shape according to the feature of the texture, but here is described a case of a square image block.
FIG. 4 is a diagram showing variation in SAD when the image block 201b is shifted by one pixel. It is conceivable that the image block 201a and the image block 201b have the highest correlation with each other when the shift amount corresponds to four pixels because the smallest SAD is obtained. Therefore, the parallax between the camera a and the camera b at the image block 201a is calculated as four pixels. In addition, multiplication of the calculated parallax and a pixel pitch size yields the parallax P in Expression (1) and the distance D to the object. In this case, although the parallax detecting apparatus can calculate a parallax approximate to the actual parallax corresponding to 3.6 pixels, the parallax detection accuracy is one pixel, and thus the parallax detecting apparatus cannot calculate the parallax at a pixel accuracy at or below the decimal point (hereinafter referred to as a sub-pixel parallax).
Methods of estimating a sub-pixel level parallax have been provided as methods of obtaining distance measurement accuracy, that is, a parallax detection resolution not at a one pixel unit but at a higher accuracy (for example, see PTL (Patent Literature) 1). For example, according to a sub-pixel parallax estimation method called equiangular linear fitting, an actual parallax is estimated at a sub-pixel level by one dimensional linear interpolation performed assuming that variation in SAD shows the same tilt θ at both the left and right sides with respect to the actual parallax as shown in FIG. 5. The following Expression (3) shows the sub-pixel parallax calculation expression, that is, the interpolation expression, in equiangular linear fitting.
                    [                  Math          .                                          ⁢          3                ]                                                                      P          =                                    P              ⁢                                                          ⁢              min                        +            d                          ⁢                                  ⁢                                            When              ⁢                                                          ⁢                              R                ⁡                                  (                  1                  )                                                      <                          R              ⁡                              (                                  -                  1                                )                                              ,                                          ⁢                      d            =                          0.5              ⁢                                                          ⁢                                                                    R                    ⁢                                          (                      1                      )                                                        -                                      R                    ⁡                                          (                                              -                        1                                            )                                                                                                            R                    ⁡                                          (                      0                      )                                                        -                                      R                    ⁡                                          (                                              -                        1                                            )                                                                                                          ⁢                                  ⁢                              The            ⁢                                                  ⁢            other            ⁢                                                  ⁢            cases                    ,                                          ⁢                      d            =                          0.5              ⁢                                                          ⁢                                                                    R                    ⁢                                          (                      1                      )                                                        -                                      R                    ⁡                                          (                                              -                        1                                            )                                                                                                            R                    ⁡                                          (                      0                      )                                                        -                                      R                    ⁡                                          (                      1                      )                                                                                                                                        (                      Expression            ⁢                                                  ⁢            3                    )                ]            
Here, P denotes a sub-pixel parallax, Pmin denotes a shift amount (an integer parallax) that yields the smallest SAD, and R(0) denotes a correlation value (SAD) at the shift amount that yields the smallest SAD. The SADs at the adjacent shift amounts are assumed to be R (−1) and R (1).
The conventional methods further include methods of calculating an actual parallax by interpolation using a high-order linear function such as a quadric or a non-linear function. In particular, in the case of an object having luminance that varies according to a linear function as shown in FIGS. 3A and 3B, variation in SAD is symmetric with respect to the actual parallax as shown in FIG. 4, and looks linear. For this reason, the parallax detecting apparatus is capable of accurately calculating the parallax corresponding to 3.6 pixels by estimating the sub-pixel parallax using equiangular linear fitting.